.\"
.\" Copyright (C) 1994-2021 Altair Engineering, Inc.
.\" For more information, contact Altair at www.altair.com.
.\"
.\" This file is part of both the OpenPBS software ("OpenPBS")
.\" and the PBS Professional ("PBS Pro") software.
.\"
.\" Open Source License Information:
.\"
.\" OpenPBS is free software. You can redistribute it and/or modify it under
.\" the terms of the GNU Affero General Public License as published by the
.\" Free Software Foundation, either version 3 of the License, or (at your
.\" option) any later version.
.\"
.\" OpenPBS is distributed in the hope that it will be useful, but WITHOUT
.\" ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
.\" FITNESS FOR A PARTICULAR PURPOSE.  See the GNU Affero General Public
.\" License for more details.
.\"
.\" You should have received a copy of the GNU Affero General Public License
.\" along with this program.  If not, see <http://www.gnu.org/licenses/>.
.\"
.\" Commercial License Information:
.\"
.\" PBS Pro is commercially licensed software that shares a common core with
.\" the OpenPBS software.  For a copy of the commercial license terms and
.\" conditions, go to: (http://www.pbspro.com/agreement.html) or contact the
.\" Altair Legal Department.
.\"
.\" Altair's dual-license business model allows companies, individuals, and
.\" organizations to create proprietary derivative works of OpenPBS and
.\" distribute them - whether embedded or bundled with other software -
.\" under a commercial license agreement.
.\"
.\" Use of Altair's trademarks, including but not limited to "PBS™",
.\" "OpenPBS®", "PBS Professional®", and "PBS Pro™" and Altair's logos is
.\" subject to Altair's trademark licensing policies.
.\"
.TH pbs_release_nodes 1B "6 May 2020" Local "PBS Professional"
.SH NAME
.B pbs_release_nodes 
\- release vnodes assigned to a PBS job
.SH SYNOPSIS
.B pbs_release_nodes
[-j <job ID>] 
.RS 18
.br
[-k (<number of hosts to keep> | 
.br
<selection of vnodes to keep>)] 
.br
<vnode> [<vnode> [<vnode>] ...]
.RE
.br
.B pbs_release_nodes 
[-j <job ID>] -a
.br
.B pbs_release_nodes
--version
.SH DESCRIPTION

You can use the 
.B pbs_release_nodes 
command to release no-longer-needed
sister hosts or vnodes assigned to a running job, before the job would
normally release them.  These vnodes are then available for use by
other jobs.  

You can specify the names of sister vnodes to be released, or you can
release all sister vnodes not on the primary execution host that are
assigned to a running job via the
.I -a 
option.

PBS can keep the numbe of sister hosts you specify, or PBS can release
all sister vnodes except for the ones you specify vi a a select
statement.  

Can be used on jobs and subjobs, but not on job arrays or ranges of
subjobs.

.B Caveats and Restrictions
.br

You can release only sister hosts or vnodes that are not on the
primary execution host.  You cannot release vnodes on the primary
execution host.

The job must be running (in the 
.I R 
state).

The pbs_release_nodes command is not supported on vnodes tied to Cray
X* series systems (vnodes whose 
.I vntype 
has the "cray_" prefix).

If cgroups support is enabled, and pbs_release_nodes is called to
release some but not all the vnodes managed by a MoM, resources on
those vnodes that are part of a cgroup are not released until the
entire cgroup is released.

You cannot release a partial host.  If you try to release some but not
all of a host, the job's 
.I exec_vnode 
attribute shows the new, smaller list of vnodes, but the pbsnodes
command will reveal that the host is still allocated to the job.

If you specify release of a vnode on which a job process is running,
that process is terminated when the vnode is released.

.B Required Privilege
.br
This command can be run by the job owner, the PBS Manager, Operator,
and Administrator, as well as root on Linux and Admin on Windows.

.SH Options to pbs_release_nodes

.IP "-a" 10
Releases all job vnodes not on the primary execution host.  Cannot be
used 
.I -k
option, or with list of vnode names.

.IP "-j <job ID>" 10
Specifies the job ID for the job or subjob whose vnode(s) are to be released.

.IP "-k <keep number> | <keep selection>" 10
Use 
.I keep number 
to specify how many sister hosts to keep.

Use 
.I keep selection 
to specify which sister vnodes to keep.  The 
.I keep selection 
is a select statement beginning with "select=" specifying which vnodes to keep.  

The primary execution host and its vnodes are not released.

For example, to release all sister hosts except 8:
.br
.B \ \ \ pbs_release_nodes -k 8
.br

To release all sister vnodes except for 4 of the ones marked with
"bigmem":

.br
.B \ \ \ pbs_release_nodes -k select=4:bigmem=true

Cannot be used with 
.I -a 
option or with vnode list argument.

.IP "(no options)" 10
With no options, pbs_release_nodes uses the value of the
.I PBS_JOBID 
environment variable as the job ID of the job whose vnodes are to be released.

.IP "--version" 10
The pbs_release_nodes command returns its PBS version information and exits.
This option can only be used alone.

.SH Operands for pbs_release_nodes
The pbs_release_nodes command can take as an operand a list of vnodes.  Format:
.br
.I <vnode name> [<vnode name> [<vnode name>] ...]
.br
Cannot be used with the 
.I -a 
option.

.SH Usage
This command can be run at the command line, or called inside a job
script, where it can use the value of the 
.I PBS_JOBID 
environment variable.

You can release any vnode that appears in the job's 
.I exec_vnode 
attribute that is not on the primary execution host.  You can release
a particular set of a job's vnodes, or you can release all of a job's
non-primary-execution-host vnodes.

To release specific vnodes:
.br
.B \ \ \  pbs_release_nodes [-j <job ID>] <vnode name> [<vnode name>] ...]

To release all of a job's vnodes that are not on the primary execution host:
.br
.B \ \ \  pbs_release_nodes [-j <job ID>] -a

To release all except a specified number of vnodes:
.br
.B \ \ \ pbs_release_nodes -k <number of sister hosts to keep>

To release all vnodes except for those in a select specification:
.br
.B \ \ \ pbs_release_nodes -k <select specification>
